{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div class=\"jumbotron\">\n",
    "    <p class=\"display-1 h1\">循环神经网络的从零开始实现</p>\n",
    "    <hr class=\"my-4\">\n",
    "    <p>主讲：李岩</p>\n",
    "    <p>管理学院</p>\n",
    "    <p>liyan@cumtb.edu.cn</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### RNN的基本思想\n",
    "\n",
    "循环神经网络（RNN）的核心思想是：\n",
    "- 使用**隐藏状态（hidden state）**来存储和传递历史信息\n",
    "- 在每个时间步，隐藏状态都会更新，包含之前所有时间步的信息\n",
    "- 这使得RNN能够处理变长序列，并捕获序列中的依赖关系\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 从语言模型的角度理解隐藏状态\n",
    "\n",
    "在语言模型中，我们的目标是估计文本序列的联合概率：\n",
    "\n",
    "$$P(x_1, x_2, \\ldots, x_T)$$\n",
    "\n",
    "根据概率的链式法则，我们可以将联合概率分解为：\n",
    "\n",
    "$$P(x_1, x_2, \\ldots, x_T) = \\prod_{t=1}^T P(x_t \\mid x_1, \\ldots, x_{t-1})$$\n",
    "\n",
    "**关键问题**：如何估计条件概率 $P(x_t \\mid x_1, \\ldots, x_{t-1})$？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**RNN的解决方案**：\n",
    "- **隐藏状态 $H_t$ 编码了历史信息**：$H_t$ 包含了从 $x_1$ 到 $x_{t-1}$ 的所有信息\n",
    "- **使用隐藏状态建模条件概率**：$P(x_t \\mid x_1, \\ldots, x_{t-1}) \\approx P(x_t \\mid H_{t-1})$\n",
    "- **隐藏状态的更新**：$H_t = f(H_{t-1}, x_t)$，其中 $f$ 是RNN的更新函数\n",
    "- **通过隐藏状态传递信息**：每个时间步，隐藏状态都会更新，将历史信息传递给未来\n",
    "\n",
    "**优势**：\n",
    "- **自动学习特征**：不需要手工设计n-gram特征\n",
    "- **捕获长距离依赖**：通过隐藏状态传递信息，可以捕获任意长度的依赖关系\n",
    "- **泛化能力**：相似上下文共享参数，提高模型的泛化能力\n",
    "- **固定维度表示**：将变长序列映射到固定维度的向量表示，便于处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### RNN的数学形式\n",
    "\n",
    "对于时间步$t$，RNN的更新公式为：\n",
    "\n",
    "$$\\mathbf{H}_t = \\phi(\\mathbf{X}_t \\mathbf{W}_{xh} + \\mathbf{H}_{t-1} \\mathbf{W}_{hh} + \\mathbf{b}_h)$$\n",
    "\n",
    "$$\\mathbf{O}_t = \\mathbf{H}_t \\mathbf{W}_{hq} + \\mathbf{b}_q$$\n",
    "\n",
    "其中：\n",
    "- $\\mathbf{X}_t$：时间步$t$的输入\n",
    "- $\\mathbf{H}_t$：时间步$t$的隐藏状态\n",
    "- $\\mathbf{O}_t$：时间步$t$的输出\n",
    "- $\\phi$：激活函数（通常使用tanh）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### RNN的计算图\n",
    "\n",
    "RNN可以展开成多个时间步的计算图：\n",
    "\n",
    "```\n",
    "t=0:  X₀ ──W_xh──┐\n",
    "                  ├→ [RNN] ──→ H₀ ──W_hq──→ O₀\n",
    "                  └─W_hh──┘ (初始H=0)\n",
    "\n",
    "t=1:  X₁ ──W_xh──┐\n",
    "      H₀ ──W_hh──┼→ [RNN] ──→ H₁ ──W_hq──→ O₁\n",
    "                  └─ (共享)\n",
    "\n",
    "t=2:  X₂ ──W_xh──┐\n",
    "      H₁ ──W_hh──┼→ [RNN] ──→ H₂ ──W_hq──→ O₂\n",
    "                  └─ (共享)\n",
    "```\n",
    "\n",
    "**关键观察**：\n",
    "- **输入组合**：每个时间步，当前输入$X_t$通过$W_{xh}$，前一时刻的隐藏状态$H_{t-1}$通过$W_{hh}$，一起输入到RNN模块\n",
    "- **权重矩阵**：\n",
    "  - $W_{xh}$：输入到隐藏状态的权重（每个时间步共享）\n",
    "  - $W_{hh}$：隐藏状态到隐藏状态的权重（每个时间步共享）\n",
    "  - $W_{hq}$：隐藏状态到输出的权重（每个时间步共享）\n",
    "- **参数共享**：所有时间步使用相同的权重矩阵$W_{xh}$、$W_{hh}$和$W_{hq}$\n",
    "- **状态传递**：$H_{t-1}$通过$W_{hh}$影响$H_t$的计算，实现信息的跨时间步传递\n",
    "- **梯度传播**：梯度通过时间反向传播（BPTT），从$H_t$流向$H_{t-1}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src='rnn_1.png' width='800'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 激活函数tanh\n",
    "\n",
    "RNN使用**tanh**作为激活函数：\n",
    "\n",
    "$$\\tanh(x) = \\frac{e^x - e^{-x}}{e^x + e^{-x}}$$\n",
    "\n",
    "**特点**：\n",
    "- 输出范围：(-1, 1)\n",
    "- 中心对称，有助于梯度流动\n",
    "- 相比sigmoid，梯度消失问题较轻\n",
    "\n",
    "**为什么不用ReLU？**\n",
    "- ReLU可能导致激活值过大\n",
    "- tanh的输出有界，更适合RNN\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### RNN需要处理文本数据，但神经网络只能处理数值向量。如何将词（离散符号）转换为向量？\n",
    "\n",
    "#### 独热编码的优缺点\n",
    "\n",
    "**优点**：\n",
    "- 实现简单\n",
    "- 每个词元有唯一的表示\n",
    "\n",
    "**缺点**：\n",
    "- 向量维度等于词汇表大小（可能很大）\n",
    "- 无法表示词之间的相似性\n",
    "- 内存占用大\n",
    "\n",
    "**现代替代方案**：\n",
    "- 词嵌入（Word Embedding）：将词映射到低维稠密向量\n",
    "- 预训练词向量：Word2Vec、GloVe等\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 语言向量化方法对比\n",
    "\n",
    "| 方法 | 向量化对象 | 向量维度 | 特点 | 应用场景 |\n",
    "|------|-----------|---------|------|---------|\n",
    "| **独热编码** | 单个词 | 词汇表大小（高维稀疏） | 简单直接，无法表示相似性 | RNN从零开始实现 |\n",
    "| **词嵌入（Word Embedding）** | 单个词 | 低维稠密（如100-300维） | 可学习，能表示语义相似性 | RNN/LSTM/Transformer |\n",
    "| **TF-IDF** | 整个文档 | 词汇表大小（高维稀疏） | 基于统计，文档级别特征 | 信息检索、文本分类 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 详细对比\n",
    "\n",
    "**1. 词嵌入（Word Embedding）**：\n",
    "- **向量化对象**：单个词\n",
    "- **向量含义**：词的语义表示\n",
    "- **特点**：\n",
    "  - 低维稠密向量（如100-300维）\n",
    "  - 语义相似的词在向量空间中距离较近\n",
    "  - 可学习的参数（通过训练优化）\n",
    "- **示例**：\n",
    "  ```\n",
    "  \"cat\" → [0.2, -0.1, 0.5, ...]  (100维向量)\n",
    "  \"dog\" → [0.3, -0.2, 0.4, ...]  (语义相似，向量也相似)\n",
    "  ```\n",
    "\n",
    "**2. TF-IDF**：\n",
    "- **向量化对象**：整个文档\n",
    "- **向量含义**：文档中词的重要性权重\n",
    "- **特点**：\n",
    "  - 高维稀疏向量（维度=词汇表大小）\n",
    "  - 基于统计特征（词频和逆文档频率）\n",
    "  - 固定公式，不可学习\n",
    "- **示例**：\n",
    "  ```\n",
    "  文档：\"The cat sat on the mat\"\n",
    "  TF-IDF向量：[0.1, 0.0, 0.3, 0.0, 0.2, ...]  (每个位置对应一个词)\n",
    "  ```\n",
    "- **应用**：文档级别的特征表示，用于文档分类、检索"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 关键区别\n",
    "\n",
    "**1. 粒度不同**：\n",
    "- **词嵌入**：词级别（word-level）\n",
    "- **TF-IDF**：文档级别（document-level）\n",
    "\n",
    "**2. 维度不同**：\n",
    "- **词嵌入**：低维稠密（学习到的紧凑表示）\n",
    "- **TF-IDF**：高维稀疏（词汇表大小）\n",
    "\n",
    "**3. 可学习性**：\n",
    "- **词嵌入**：可学习（通过反向传播优化）\n",
    "- **TF-IDF**：固定公式（基于统计，不可学习）\n",
    "\n",
    "**4. 语义理解**：\n",
    "- **词嵌入**：能捕获语义相似性（\"cat\"和\"dog\"向量相似）\n",
    "- **TF-IDF**：只考虑词频，无法表示语义关系"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 在RNN中的应用\n",
    "\n",
    "**传统方法（从零开始）**：\n",
    "```python\n",
    "# 使用独热编码\n",
    "X = F.one_hot(X, vocab_size)  # 高维稀疏\n",
    "H = tanh(X @ W_xh + H @ W_hh)  # W_xh很大：(vocab_size, num_hiddens)\n",
    "```\n",
    "\n",
    "**现代方法（使用词嵌入）**：\n",
    "```python\n",
    "# 使用词嵌入层\n",
    "embedding = nn.Embedding(vocab_size, embed_size)  # embed_size << vocab_size\n",
    "X = embedding(X)  # 低维稠密\n",
    "H = tanh(X @ W_xh + H @ W_hh)  # W_xh较小：(embed_size, num_hiddens)\n",
    "```\n",
    "\n",
    "**TF-IDF在RNN中**：\n",
    "- 通常**不直接用于RNN**（因为RNN需要序列输入，而TF-IDF是文档级别）\n",
    "- 但可以用于**预处理**或**特征工程**（如文档分类任务）\n",
    "\n",
    "#### 总结\n",
    "\n",
    "- **都是向量化方法**：将语言转换为数值向量\n",
    "- **不同层面**：词嵌入是词级别，TF-IDF是文档级别\n",
    "- **不同目的**：词嵌入用于序列建模（RNN/LSTM），TF-IDF用于文档表示（检索/分类）\n",
    "- **在RNN中**：词嵌入是标准方法，TF-IDF通常不直接使用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### RNN的参数\n",
    "\n",
    "RNN需要学习的参数包括：\n",
    "- $\\mathbf{W}_{xh}$：输入到隐藏层的权重矩阵\n",
    "- $\\mathbf{W}_{hh}$：隐藏层到隐藏层的权重矩阵（循环权重）\n",
    "- $\\mathbf{b}_h$：隐藏层的偏置\n",
    "- $\\mathbf{W}_{hq}$：隐藏层到输出层的权重矩阵\n",
    "- $\\mathbf{b}_q$：输出层的偏置\n",
    "\n",
    "**关键特点**：参数在所有时间步之间**共享**，这使得模型可以处理任意长度的序列。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:25:29.064904Z",
     "iopub.status.busy": "2022-07-31T02:25:29.064655Z",
     "iopub.status.idle": "2022-07-31T02:25:29.332873Z",
     "shell.execute_reply": "2022-07-31T02:25:29.332112Z"
    },
    "origin_pos": 4,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading ..\\data\\timemachine.txt from http://d2l-data.s3-accelerate.amazonaws.com/timemachine.txt...\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import math\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "from d2l import torch as d2l\n",
    "\n",
    "batch_size, num_steps = 32, 35\n",
    "train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 数据形状说明\n",
    "\n",
    "对于RNN，输入数据的形状需要特别注意：\n",
    "\n",
    "**形状转换过程**：\n",
    "- **原始形状**：`(batch_size, num_steps)` - 二维张量\n",
    "- **转置后**：`(num_steps, batch_size)` - 便于按时间步迭代\n",
    "- **独热编码后**：`(num_steps, batch_size, vocab_size)` - 三维张量\n",
    "\n",
    "**为什么需要转置？**\n",
    "- 便于按时间步顺序处理\n",
    "- 每个时间步处理一个`(batch_size, vocab_size)`的矩阵\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 初始化模型参数\n",
    "\n",
    "### 参数初始化的重要性\n",
    "\n",
    "在深度学习中，参数初始化对模型训练至关重要：\n",
    "- **好的初始化**：有助于梯度流动，加快收敛\n",
    "- **坏的初始化**：可能导致梯度消失或梯度爆炸\n",
    "\n",
    "对于RNN，我们使用**小随机值**初始化权重，偏置初始化为0。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 初始化隐藏状态\n",
    "\n",
    "在RNN中，第一个时间步没有前一个隐藏状态，因此需要初始化。\n",
    "\n",
    "**init_rnn_state函数**：\n",
    "- 返回一个全零的隐藏状态张量\n",
    "- 形状：`(batch_size, num_hiddens)`\n",
    "- 使用元组包装，便于后续扩展（如LSTM需要多个状态）\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "origin_pos": 20,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def init_rnn_state(batch_size, num_hiddens, device):\n",
    "    return (torch.zeros((batch_size, num_hiddens), device=device), )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:25:29.358465Z",
     "iopub.status.busy": "2022-07-31T02:25:29.358200Z",
     "iopub.status.idle": "2022-07-31T02:25:29.364048Z",
     "shell.execute_reply": "2022-07-31T02:25:29.363372Z"
    },
    "origin_pos": 16,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def get_params(vocab_size, num_hiddens, device):\n",
    "    num_inputs = num_outputs = vocab_size\n",
    "\n",
    "    def normal(shape):\n",
    "        return torch.randn(size=shape, device=device) * 0.01\n",
    "\n",
    "    W_xh = normal((num_inputs, num_hiddens))\n",
    "    W_hh = normal((num_hiddens, num_hiddens))\n",
    "    b_h = torch.zeros(num_hiddens, device=device)\n",
    "    W_hq = normal((num_hiddens, num_outputs))\n",
    "    b_q = torch.zeros(num_outputs, device=device)\n",
    "    params = [W_xh, W_hh, b_h, W_hq, b_q]\n",
    "    for param in params:\n",
    "        param.requires_grad_(True)\n",
    "    return params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RNN模型\n",
    "\n",
    "### RNNModelScratch类\n",
    "\n",
    "这个类封装了RNN模型的所有功能：\n",
    "\n",
    "**设计模式**：\n",
    "- 使用函数式编程：将`get_params`、`init_state`、`forward_fn`作为参数传入\n",
    "- 提高了代码的灵活性和可复用性\n",
    "\n",
    "**关键方法**：\n",
    "- `__init__`：初始化模型，创建参数\n",
    "- `__call__`：前向传播，将输入转换为独热编码并调用前向函数\n",
    "- `begin_state`：返回初始隐藏状态\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:25:29.373996Z",
     "iopub.status.busy": "2022-07-31T02:25:29.373748Z",
     "iopub.status.idle": "2022-07-31T02:25:29.378899Z",
     "shell.execute_reply": "2022-07-31T02:25:29.378276Z"
    },
    "origin_pos": 24,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def rnn(inputs, state, params):\n",
    "    W_xh, W_hh, b_h, W_hq, b_q = params\n",
    "    H, = state\n",
    "    outputs = []\n",
    "    for X in inputs:\n",
    "        H = torch.tanh(torch.mm(X, W_xh) + torch.mm(H, W_hh) + b_h)\n",
    "        Y = torch.mm(H, W_hq) + b_q\n",
    "        outputs.append(Y)\n",
    "    return torch.cat(outputs, dim=0), (H,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "origin_pos": 28,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "class RNNModelScratch:\n",
    "    \"\"\"从零开始实现的循环神经网络模型\"\"\"\n",
    "    def __init__(self, vocab_size, num_hiddens, device,\n",
    "                 get_params, init_state, forward_fn):\n",
    "        self.vocab_size, self.num_hiddens = vocab_size, num_hiddens\n",
    "        self.params = get_params(vocab_size, num_hiddens, device)\n",
    "        self.init_state, self.forward_fn = init_state, forward_fn\n",
    "\n",
    "    def __call__(self, X, state):\n",
    "        X = F.one_hot(X.T, self.vocab_size).type(torch.float32)\n",
    "        return self.forward_fn(X, state, self.params)\n",
    "\n",
    "    def begin_state(self, batch_size, device):\n",
    "        return self.init_state(batch_size, self.num_hiddens, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:25:29.390445Z",
     "iopub.status.busy": "2022-07-31T02:25:29.390211Z",
     "iopub.status.idle": "2022-07-31T02:25:33.180499Z",
     "shell.execute_reply": "2022-07-31T02:25:33.179572Z"
    },
    "origin_pos": 32,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([10, 28]), 1, torch.Size([2, 512]))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_hiddens = 512\n",
    "net = RNNModelScratch(len(vocab), num_hiddens, d2l.try_gpu(), get_params,\n",
    "                      init_rnn_state, rnn)\n",
    "state = net.begin_state(X.shape[0], d2l.try_gpu())\n",
    "Y, new_state = net(X.to(d2l.try_gpu()), state)\n",
    "Y.shape, len(new_state), new_state[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 预测\n",
    "\n",
    "### 文本生成\n",
    "\n",
    "`predict_ch8`函数用于生成文本：\n",
    "\n",
    "**工作原理**：\n",
    "1. **预热阶段**：使用给定的前缀（prefix）更新隐藏状态\n",
    "2. **生成阶段**：基于当前隐藏状态预测下一个字符\n",
    "3. **自回归**：将预测的字符作为下一个时间步的输入\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:25:33.193821Z",
     "iopub.status.busy": "2022-07-31T02:25:33.193174Z",
     "iopub.status.idle": "2022-07-31T02:25:33.207762Z",
     "shell.execute_reply": "2022-07-31T02:25:33.207070Z"
    },
    "origin_pos": 39,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'time traveller ehnfts cfm'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def predict_ch8(prefix, num_preds, net, vocab, device):\n",
    "    \"\"\"在prefix后面生成新字符\"\"\"\n",
    "    state = net.begin_state(batch_size=1, device=device)\n",
    "    outputs = [vocab[prefix[0]]]\n",
    "    get_input = lambda: torch.tensor([outputs[-1]], device=device).reshape((1, 1))\n",
    "    for y in prefix[1:]:\n",
    "        _, state = net(get_input(), state)\n",
    "        outputs.append(vocab[y])\n",
    "    for _ in range(num_preds):\n",
    "        y, state = net(get_input(), state)\n",
    "        outputs.append(int(y.argmax(dim=1).reshape(1)))\n",
    "    return ''.join([vocab.idx_to_token[i] for i in outputs])\n",
    "\n",
    "predict_ch8('time traveller ', 10, net, vocab, d2l.try_gpu())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 自回归生成\n",
    "\n",
    "文本生成是**自回归（Autoregressive）**过程：\n",
    "\n",
    "- 模型预测下一个字符的概率分布\n",
    "- 选择概率最大的字符（greedy decoding）\n",
    "- 将预测的字符作为下一个时间步的输入\n",
    "- 重复此过程，生成完整序列\n",
    "\n",
    "**其他解码策略**：\n",
    "- **采样（Sampling）**：根据概率分布随机采样\n",
    "- **束搜索（Beam Search）**：保留多个候选序列\n",
    "- **温度采样（Temperature Sampling）**：控制生成的随机性\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 梯度裁剪\n",
    "\n",
    "### 为什么需要梯度裁剪？\n",
    "\n",
    "RNN在训练过程中容易出现**梯度爆炸**问题：\n",
    "\n",
    "**原因**：\n",
    "- RNN在每个时间步都使用相同的权重矩阵\n",
    "- 梯度通过时间反向传播（BPTT）\n",
    "- 如果权重矩阵的最大特征值>1，梯度会指数增长\n",
    "\n",
    "### 梯度裁剪的数学形式\n",
    "\n",
    "$$\\mathbf{g} \\leftarrow \\min\\left(1, \\frac{\\theta}{\\|\\mathbf{g}\\|}\\right) \\mathbf{g}$$\n",
    "\n",
    "其中：\n",
    "- $\\mathbf{g}$：梯度向量\n",
    "- $\\theta$：裁剪阈值（通常设为1）\n",
    "- $\\|\\mathbf{g}\\|$：梯度的L2范数\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### grad_clipping函数实现\n",
    "\n",
    "**实现步骤**：\n",
    "1. 收集所有需要梯度的参数\n",
    "2. 计算所有参数梯度的L2范数\n",
    "3. 如果范数超过阈值，按比例缩放所有梯度\n",
    "\n",
    "**关键点**：\n",
    "- 使用`in-place`操作（`[:]`）直接修改梯度\n",
    "- 支持两种模型类型：`nn.Module`和自定义模型\n",
    "- 保持梯度的方向不变，只改变大小\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:25:33.210992Z",
     "iopub.status.busy": "2022-07-31T02:25:33.210523Z",
     "iopub.status.idle": "2022-07-31T02:25:33.216280Z",
     "shell.execute_reply": "2022-07-31T02:25:33.215362Z"
    },
    "origin_pos": 43,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def grad_clipping(net, theta):\n",
    "    \"\"\"裁剪梯度\"\"\"\n",
    "    if isinstance(net, nn.Module):\n",
    "        params = [p for p in net.parameters() if p.requires_grad]\n",
    "    else:\n",
    "        params = net.params\n",
    "    norm = torch.sqrt(sum(torch.sum((p.grad ** 2)) for p in params))\n",
    "    if norm > theta:\n",
    "        for param in params:\n",
    "            param.grad[:] *= theta / norm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 训练\n",
    "\n",
    "### 训练一个迭代周期\n",
    "\n",
    "`train_epoch_ch8`函数实现了一个完整的训练迭代周期：\n",
    "\n",
    "**关键步骤**：\n",
    "1. **初始化隐藏状态**：随机采样或顺序采样\n",
    "2. **前向传播**：计算预测输出和损失\n",
    "3. **反向传播**：计算梯度，应用梯度裁剪，更新参数\n",
    "4. **记录指标**：计算困惑度和训练速度\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:25:33.219671Z",
     "iopub.status.busy": "2022-07-31T02:25:33.219190Z",
     "iopub.status.idle": "2022-07-31T02:25:33.227728Z",
     "shell.execute_reply": "2022-07-31T02:25:33.226847Z"
    },
    "origin_pos": 47,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def train_epoch_ch8(net, train_iter, loss, updater, device, use_random_iter):\n",
    "    \"\"\"训练网络一个迭代周期（定义见第8章）\"\"\"\n",
    "    state, timer = None, d2l.Timer()\n",
    "    metric = d2l.Accumulator(2)\n",
    "    for X, Y in train_iter:\n",
    "        if state is None or use_random_iter:\n",
    "            state = net.begin_state(batch_size=X.shape[0], device=device)\n",
    "        else:\n",
    "            if isinstance(net, nn.Module) and not isinstance(state, tuple):\n",
    "                state.detach_()\n",
    "            else:\n",
    "                for s in state:\n",
    "                    s.detach_()\n",
    "        y = Y.T.reshape(-1)\n",
    "        X, y = X.to(device), y.to(device)\n",
    "        y_hat, state = net(X, state)\n",
    "        l = loss(y_hat, y.long()).mean()\n",
    "        if isinstance(updater, torch.optim.Optimizer):\n",
    "            updater.zero_grad()\n",
    "            l.backward()\n",
    "            grad_clipping(net, 1)\n",
    "            updater.step()\n",
    "        else:\n",
    "            l.backward()\n",
    "            grad_clipping(net, 1)\n",
    "            updater(batch_size=1)\n",
    "        metric.add(l * y.numel(), y.numel())\n",
    "    return math.exp(metric[0] / metric[1]), metric[1] / timer.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### train_ch8函数\n",
    "\n",
    "这是完整的训练函数，包含：\n",
    "\n",
    "**功能**：\n",
    "- 初始化损失函数和优化器\n",
    "- 创建可视化工具（困惑度曲线）\n",
    "- 训练多个epoch\n",
    "- 定期生成文本样本，观察模型效果\n",
    "\n",
    "**训练监控**：\n",
    "- 每10个epoch打印一次生成的文本\n",
    "- 绘制困惑度曲线\n",
    "- 训练结束后打印最终结果\n",
    "\n",
    "**设计特点**：\n",
    "- 同时支持从零实现的模型和`nn.Module`模型\n",
    "- 灵活选择随机采样或顺序采样\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "origin_pos": 51,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def train_ch8(net, train_iter, vocab, lr, num_epochs, device,\n",
    "              use_random_iter=False):\n",
    "    \"\"\"训练模型（定义见第8章）\"\"\"\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "    animator = d2l.Animator(xlabel='epoch', ylabel='perplexity',\n",
    "                            legend=['train'], xlim=[10, num_epochs])\n",
    "    if isinstance(net, nn.Module):\n",
    "        updater = torch.optim.SGD(net.parameters(), lr)\n",
    "    else:\n",
    "        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)\n",
    "    predict = lambda prefix: predict_ch8(prefix, 50, net, vocab, device)\n",
    "    for epoch in range(num_epochs):\n",
    "        ppl, speed = train_epoch_ch8(\n",
    "            net, train_iter, loss, updater, device, use_random_iter)\n",
    "        if (epoch + 1) % 10 == 0:\n",
    "            print(predict('time traveller'))\n",
    "            animator.add(epoch + 1, [ppl])\n",
    "    print(f'困惑度 {ppl:.1f}, {speed:.1f} 词元/秒 {str(device)}')\n",
    "    print(predict('time traveller'))\n",
    "    print(predict('traveller'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:25:33.241737Z",
     "iopub.status.busy": "2022-07-31T02:25:33.241105Z",
     "iopub.status.idle": "2022-07-31T02:26:40.718409Z",
     "shell.execute_reply": "2022-07-31T02:26:40.717797Z"
    },
    "origin_pos": 54,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "困惑度 1.0, 79122.0 词元/秒 cuda:0\n",
      "time traveller with a slight accession ofcheerfulness really thi\n",
      "travelleryou can show black is white by argument said filby\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"262.1875pt\" height=\"180.65625pt\" viewBox=\"0 0 262.1875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-07-31T02:26:40.681588</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 262.1875 180.65625 \n",
       "L 262.1875 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 143.1 \n",
       "L 245.44375 143.1 \n",
       "L 245.44375 7.2 \n",
       "L 50.14375 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 86.015179 143.1 \n",
       "L 86.015179 7.2 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"meef6c204ac\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#meef6c204ac\" x=\"86.015179\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(76.471429 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 125.872321 143.1 \n",
       "L 125.872321 7.2 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#meef6c204ac\" x=\"125.872321\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(116.328571 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 165.729464 143.1 \n",
       "L 165.729464 7.2 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#meef6c204ac\" x=\"165.729464\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(156.185714 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 205.586607 143.1 \n",
       "L 205.586607 7.2 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#meef6c204ac\" x=\"205.586607\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(196.042857 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 245.44375 143.1 \n",
       "L 245.44375 7.2 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#meef6c204ac\" x=\"245.44375\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(235.9 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(132.565625 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 50.14375 122.572452 \n",
       "L 245.44375 122.572452 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m3ce896ce03\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ce896ce03\" x=\"50.14375\" y=\"122.572452\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 2.5 -->\n",
       "      <g transform=\"translate(27.240625 126.371671)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 50.14375 98.152391 \n",
       "L 245.44375 98.152391 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ce896ce03\" x=\"50.14375\" y=\"98.152391\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 5.0 -->\n",
       "      <g transform=\"translate(27.240625 101.95161)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 50.14375 73.73233 \n",
       "L 245.44375 73.73233 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ce896ce03\" x=\"50.14375\" y=\"73.73233\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 7.5 -->\n",
       "      <g transform=\"translate(27.240625 77.531549)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-37\" d=\"M 525 4666 \n",
       "L 3525 4666 \n",
       "L 3525 4397 \n",
       "L 1831 0 \n",
       "L 1172 0 \n",
       "L 2766 4134 \n",
       "L 525 4134 \n",
       "L 525 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-37\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 50.14375 49.312269 \n",
       "L 245.44375 49.312269 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ce896ce03\" x=\"50.14375\" y=\"49.312269\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 10.0 -->\n",
       "      <g transform=\"translate(20.878125 53.111488)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 50.14375 24.892208 \n",
       "L 245.44375 24.892208 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ce896ce03\" x=\"50.14375\" y=\"24.892208\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 12.5 -->\n",
       "      <g transform=\"translate(20.878125 28.691427)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 100.276562)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 50.14375 13.377273 \n",
       "L 54.129464 42.499479 \n",
       "L 58.115179 52.912978 \n",
       "L 62.100893 58.593644 \n",
       "L 66.086607 63.311957 \n",
       "L 70.072321 67.411097 \n",
       "L 74.058036 68.685211 \n",
       "L 78.04375 71.684165 \n",
       "L 82.029464 73.15806 \n",
       "L 86.015179 75.892363 \n",
       "L 90.000893 79.353406 \n",
       "L 93.986607 80.910004 \n",
       "L 97.972321 83.16332 \n",
       "L 101.958036 87.772112 \n",
       "L 105.94375 89.142841 \n",
       "L 109.929464 96.31637 \n",
       "L 113.915179 101.053922 \n",
       "L 117.900893 107.539587 \n",
       "L 121.886607 112.988293 \n",
       "L 125.872321 119.223677 \n",
       "L 129.858036 121.729995 \n",
       "L 133.84375 125.787051 \n",
       "L 137.829464 128.576681 \n",
       "L 141.815179 130.606282 \n",
       "L 145.800893 131.588404 \n",
       "L 149.786607 133.007219 \n",
       "L 153.772321 133.359002 \n",
       "L 157.758036 134.015462 \n",
       "L 161.74375 134.320011 \n",
       "L 165.729464 134.688864 \n",
       "L 169.715179 134.992905 \n",
       "L 173.700893 136.09362 \n",
       "L 177.686607 136.188532 \n",
       "L 181.672321 135.865298 \n",
       "L 185.658036 136.209251 \n",
       "L 189.64375 136.568029 \n",
       "L 193.629464 136.474669 \n",
       "L 197.615179 135.363953 \n",
       "L 201.600893 136.281153 \n",
       "L 205.586607 136.562972 \n",
       "L 209.572321 136.288907 \n",
       "L 213.558036 136.69562 \n",
       "L 217.54375 136.472615 \n",
       "L 221.529464 136.841775 \n",
       "L 225.515179 136.876423 \n",
       "L 229.500893 136.823047 \n",
       "L 233.486607 136.78962 \n",
       "L 237.472321 136.918469 \n",
       "L 241.458036 136.777658 \n",
       "L 245.44375 136.922727 \n",
       "\" clip-path=\"url(#p68b87d4c2e)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 143.1 \n",
       "L 50.14375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 245.44375 143.1 \n",
       "L 245.44375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 143.1 \n",
       "L 245.44375 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 7.2 \n",
       "L 245.44375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 183.16875 29.878125 \n",
       "L 238.44375 29.878125 \n",
       "Q 240.44375 29.878125 240.44375 27.878125 \n",
       "L 240.44375 14.2 \n",
       "Q 240.44375 12.2 238.44375 12.2 \n",
       "L 183.16875 12.2 \n",
       "Q 181.16875 12.2 181.16875 14.2 \n",
       "L 181.16875 27.878125 \n",
       "Q 181.16875 29.878125 183.16875 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_22\">\n",
       "     <path d=\"M 185.16875 20.298437 \n",
       "L 195.16875 20.298437 \n",
       "L 205.16875 20.298437 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(213.16875 23.798437)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p68b87d4c2e\">\n",
       "   <rect x=\"50.14375\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_epochs, lr = 500, 1\n",
    "train_ch8(net, train_iter, vocab, lr, num_epochs, d2l.try_gpu())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:40.721600Z",
     "iopub.status.busy": "2022-07-31T02:26:40.721105Z",
     "iopub.status.idle": "2022-07-31T02:27:51.596800Z",
     "shell.execute_reply": "2022-07-31T02:27:51.595904Z"
    },
    "origin_pos": 57,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "困惑度 1.4, 75593.4 词元/秒 cuda:0\n",
      "time traveller smiled rou gannet of opracinbutway ohacknat difte\n",
      "travellerit s against reason said filbywan a cube that does\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"262.1875pt\" height=\"180.65625pt\" viewBox=\"0 0 262.1875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-07-31T02:27:51.558852</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 262.1875 180.65625 \n",
       "L 262.1875 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 143.1 \n",
       "L 245.44375 143.1 \n",
       "L 245.44375 7.2 \n",
       "L 50.14375 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 86.015179 143.1 \n",
       "L 86.015179 7.2 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m90308742b0\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90308742b0\" x=\"86.015179\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(76.471429 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 125.872321 143.1 \n",
       "L 125.872321 7.2 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90308742b0\" x=\"125.872321\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(116.328571 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 165.729464 143.1 \n",
       "L 165.729464 7.2 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90308742b0\" x=\"165.729464\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(156.185714 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 205.586607 143.1 \n",
       "L 205.586607 7.2 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90308742b0\" x=\"205.586607\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(196.042857 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 245.44375 143.1 \n",
       "L 245.44375 7.2 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90308742b0\" x=\"245.44375\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(235.9 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(132.565625 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 50.14375 125.755335 \n",
       "L 245.44375 125.755335 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"maa93787fd2\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#maa93787fd2\" x=\"50.14375\" y=\"125.755335\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 2.5 -->\n",
       "      <g transform=\"translate(27.240625 129.554553)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 50.14375 100.567672 \n",
       "L 245.44375 100.567672 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maa93787fd2\" x=\"50.14375\" y=\"100.567672\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 5.0 -->\n",
       "      <g transform=\"translate(27.240625 104.36689)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 50.14375 75.380009 \n",
       "L 245.44375 75.380009 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maa93787fd2\" x=\"50.14375\" y=\"75.380009\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 7.5 -->\n",
       "      <g transform=\"translate(27.240625 79.179227)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-37\" d=\"M 525 4666 \n",
       "L 3525 4666 \n",
       "L 3525 4397 \n",
       "L 1831 0 \n",
       "L 1172 0 \n",
       "L 2766 4134 \n",
       "L 525 4134 \n",
       "L 525 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-37\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 50.14375 50.192346 \n",
       "L 245.44375 50.192346 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maa93787fd2\" x=\"50.14375\" y=\"50.192346\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 10.0 -->\n",
       "      <g transform=\"translate(20.878125 53.991564)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 50.14375 25.004683 \n",
       "L 245.44375 25.004683 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maa93787fd2\" x=\"50.14375\" y=\"25.004683\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 12.5 -->\n",
       "      <g transform=\"translate(20.878125 28.803901)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 100.276562)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 50.14375 13.377273 \n",
       "L 54.129464 43.196848 \n",
       "L 58.115179 53.771619 \n",
       "L 62.100893 59.764072 \n",
       "L 66.086607 62.320552 \n",
       "L 70.072321 66.268996 \n",
       "L 74.058036 69.567441 \n",
       "L 78.04375 71.025284 \n",
       "L 82.029464 73.737773 \n",
       "L 86.015179 76.315255 \n",
       "L 90.000893 78.350368 \n",
       "L 93.986607 81.279651 \n",
       "L 97.972321 81.971464 \n",
       "L 101.958036 83.395869 \n",
       "L 105.94375 88.088931 \n",
       "L 109.929464 90.651147 \n",
       "L 113.915179 94.355451 \n",
       "L 117.900893 97.960361 \n",
       "L 121.886607 103.039973 \n",
       "L 125.872321 107.636315 \n",
       "L 129.858036 112.141256 \n",
       "L 133.84375 117.087011 \n",
       "L 137.829464 119.313921 \n",
       "L 141.815179 123.657669 \n",
       "L 145.800893 125.757129 \n",
       "L 149.786607 129.361869 \n",
       "L 153.772321 127.591374 \n",
       "L 157.758036 131.474963 \n",
       "L 161.74375 130.551435 \n",
       "L 165.729464 131.791668 \n",
       "L 169.715179 134.166384 \n",
       "L 173.700893 131.395543 \n",
       "L 177.686607 133.779237 \n",
       "L 181.672321 133.914473 \n",
       "L 185.658036 132.494682 \n",
       "L 189.64375 134.526267 \n",
       "L 193.629464 135.295217 \n",
       "L 197.615179 134.878652 \n",
       "L 201.600893 134.97071 \n",
       "L 205.586607 135.76251 \n",
       "L 209.572321 134.049217 \n",
       "L 213.558036 134.88155 \n",
       "L 217.54375 135.343176 \n",
       "L 221.529464 134.628353 \n",
       "L 225.515179 136.024766 \n",
       "L 229.500893 136.922727 \n",
       "L 233.486607 135.017855 \n",
       "L 237.472321 136.836361 \n",
       "L 241.458036 135.849417 \n",
       "L 245.44375 136.597796 \n",
       "\" clip-path=\"url(#p79a95f2baf)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 143.1 \n",
       "L 50.14375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 245.44375 143.1 \n",
       "L 245.44375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 143.1 \n",
       "L 245.44375 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 7.2 \n",
       "L 245.44375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 183.16875 29.878125 \n",
       "L 238.44375 29.878125 \n",
       "Q 240.44375 29.878125 240.44375 27.878125 \n",
       "L 240.44375 14.2 \n",
       "Q 240.44375 12.2 238.44375 12.2 \n",
       "L 183.16875 12.2 \n",
       "Q 181.16875 12.2 181.16875 14.2 \n",
       "L 181.16875 27.878125 \n",
       "Q 181.16875 29.878125 183.16875 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_22\">\n",
       "     <path d=\"M 185.16875 20.298437 \n",
       "L 195.16875 20.298437 \n",
       "L 205.16875 20.298437 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(213.16875 23.798437)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p79a95f2baf\">\n",
       "   <rect x=\"50.14375\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "net = RNNModelScratch(len(vocab), num_hiddens, d2l.try_gpu(), get_params,\n",
    "                      init_rnn_state, rnn)\n",
    "train_ch8(net, train_iter, vocab, lr, num_epochs, d2l.try_gpu(),\n",
    "          use_random_iter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 总结与思考\n",
    "\n",
    "### 关键概念回顾\n",
    "\n",
    "1. **RNN前向传播**：通过隐藏状态传递历史信息\n",
    "2. **参数共享**：所有时间步共享相同的权重矩阵\n",
    "3. **梯度裁剪**：防止梯度爆炸，稳定训练\n",
    "4. **隐藏状态管理**：随机采样vs顺序采样的不同处理方式\n",
    "\n",
    "### RNN的优缺点\n",
    "\n",
    "**优点**：\n",
    "- 能够处理变长序列\n",
    "- 参数共享，模型参数少\n",
    "- 理论上可以捕获任意长度的依赖\n",
    "\n",
    "**缺点**：\n",
    "- 实际中难以学习长期依赖（梯度消失）\n",
    "- 计算效率低（无法并行化）\n",
    "- 容易出现梯度爆炸或梯度消失\n",
    "\n",
    "### 思考问题\n",
    "\n",
    "1. 为什么RNN需要梯度裁剪？梯度消失和梯度爆炸哪个更常见？\n",
    "2. 顺序采样和随机采样对RNN训练有什么不同影响？\n",
    "3. 如何理解RNN的参数共享？它带来了什么好处和限制？\n",
    "4. RNN的隐藏状态维度如何影响模型性能？\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "rise": {
   "autolaunch": false,
   "enable_chalkboard": true,
   "overlay": "",
   "scroll": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
